support using local image tarballs with FROM #52

vito · 2021-01-08T21:40:30Z

This adds 'image args', configurable with IMAGE_ARG_* - similar to BUILD_ARG_*, only the value points to an image tarball.

The image tarball will be loaded and served by a local registry, and a reference to the image in the local registry will be provided as the build arg.

To use this, you must modify your Dockerfile like so:

ARG base_image=ubuntu
FROM ${base_image}

Then, when running oci-build-task, specify:

params:
  IMAGE_ARG_base_image: ubuntu/image.tar

This will remain forward compatible if we ever switch to Kaniko (#46), which would also require using build args as Kaniko image caching requires a full digest to be specified in FROM.

fixes #1
closes #2
closes #3
closes #14

This adds 'image args', configurable with IMAGE_ARG_* - similar to BUILD_ARG_*, only the value points to an image tarball. The image tarball will be loaded and served by a local registry, and a reference to the image in the local registry will be provided as the build arg. To use this, you must modify your Dockerfile like so: ARG base_image=ubuntu FROM ${base_image} Then, when running oci-build-task, specify: params: IMAGE_ARG_base_image: ubuntu/image.tar This will remain forward compatible if we ever switch to Kaniko (#46), which would also require using build args as Kaniko image caching requires a full digest to be specified in FROM. fixes #1 closes #2 closes #3 closes #14 Signed-off-by: Alex Suraci <suraci.alex@gmail.com>

vito · 2021-01-11T04:10:07Z

just FYI: seeing issues with this, reproducible in a local test but haven't found the root cause yet. building works fine but the resulting image seems corrupt in some way (fails with unexpected EOF when trying to unpack). will ship this as soon as I figure that out.

vito · 2021-01-12T05:19:20Z

just FYI: seeing issues with this, reproducible in a local test but haven't found the root cause yet. building works fine but the resulting image seems corrupt in some way (fails with unexpected EOF when trying to unpack). will ship this as soon as I figure that out.

fixed in f4c893c

cpadgett-comcast · 2021-08-04T19:15:15Z

For those who end up here, there are some gotchas to the solution above:

The variable name used for FROM ${base_image} must be all lowercase, otherwise the build will fail with an error similar to failed to parse stage name "localhost:34785/DOCKERFILE_FROM_IMAGE": invalid reference format: repository name must be lowercase
The FROM image being used locally must specify the parameter format: oci, otherwise the build will fail with an error similar to failed to build: create local image registry: image from path: open base-image/image.tar: no such file or directory
The FROM image must be an input to the task, otherwise you will get the same error as 2 above.
The outputs for the image building task must be image, otherwise the build will fail with an error similar to no files match glob 'some-mistaken-output-image-name/image.tar'

Example pipeline

`build-docker-image.yaml`

---
platform: linux

image_resource:
  type: registry-image
  source:
    repository: concourse/oci-build-task

inputs:
- name: image-src
  path: .
- name: docker-from-image

outputs:
- name: image

run:
  path: build

`Dockerfile`

ARG  dockerfile_from_image=repository/url/path/image:tag
FROM ${dockerfile_from_image}

<more-dockerfile-stuff>

`pipeline.yaml`

resources:
- name: concourse-tasks-repo
  type: git
  source:
    uri: ((concourse_tasks_repo))
    branch: ((concourse_tasks_release_branch))

- name: tools-image-source-repo
  type: git
  source:
    uri: ((tools_image_src_repo))
    branch: ((tools_image_src_release_branch))

- name: tools-from-image
  type: registry-image
  source:
    repository: "((tools_from_image_docker_repo))"
    username: ((dockerregistry.username))  # store your secrets in a secrets manager like Vault
    password: ((dockerregistry.password))  # store your secrets in a secrets manager like Vault

- name: tools-image
  type: registry-image
  source:
    repository: "((tools_image_docker_repo))"
    username: ((dockerregistry.username))  # store your secrets in a secrets manager like Vault
    password: ((dockerregistry.password))  # store your secrets in a secrets manager like Vault

jobs:
- name: build-and-push-tools-image
  plan:
  - in_parallel:
      fail_fast: true
      steps:
      - get: concourse-tasks-repo
        trigger: true
      - get: tools-from-image
        params:
          format: oci
        trigger: true

  - task: build-tools-image
    privileged: true
    file: "concourse-tasks-repo/build-docker-image.yml"
    input_mapping:
      docker-from-image: tools-from-image
      image-src: tools-image-source-repo
    output_mapping:
      image: built-tools-image
    params:
      IMAGE_ARG_dockerfile_from_image: docker-from-image/image.tar

  - put: tools-image
    params:
      image: built-tools-image/image.tar

This commit updates the `build-images` pipeline to specify the `govuk-ruby-2.7.X` image as the base image for building app images. There are a [number of different ways to do this](concourse/oci-build-task#14) but this commit uses makes use of the [`IMAGE_ARG_*` param as part of the `vito/oci-build-task` resource](concourse/oci-build-task#52) which allows us to use image resources already defined in the pipeline within our Dockerfiles, where the base image is served from a registry local to the pipeline. It does this by bringing in these images as resources, namespaced under separate ECR repositories, then passing in the appropriate image to the build process via the `IMAGE_ARG_base_image` param (reflected in app `Dockerfile`s); the `FROM` command in said `Dockerfile`s then point to the correct ECR repository for the `govuk-ruby-2.7.X` base image (rather than to Docker Hub).

See https://github.com/concourse/oci-build-task/pull/52\#issuecomment-892907161 for an example.

Source: concourse/oci-build-task#52

This was referenced Jan 8, 2021

Support insecure registries in FROM #3

Closed

Configurable CA certs for fetching FROM images #2

Closed

Support http basic auth credentials for FROM repo. #14

Closed

vito merged commit 38fb352 into master Jan 8, 2021

vito deleted the image-args branch January 12, 2021 16:52

ringods mentioned this pull request Mar 2, 2021

To build an image which base is in ECR and needs additional authentication #57

Closed

suhlig referenced this pull request in suhlig/concourse-task-library Aug 1, 2022

Add input for optional base image

725edae

See https://github.com/concourse/oci-build-task/pull/52\#issuecomment-892907161 for an example.

holgerstolzenberg mentioned this pull request Sep 28, 2022

How do I use a private registry in Dockerfile FROM directive? #70

Closed

imsodin mentioned this pull request Jan 17, 2023

The IMAGE_ARG_* mechanism doesn't work with multi-arch builds #110

Open

jameshochadel added a commit to cloud-gov/common-pipelines that referenced this pull request Aug 4, 2023

Try params suggested in original GitHub PR

8e7cf7b

Source: concourse/oci-build-task#52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support using local image tarballs with FROM #52

support using local image tarballs with FROM #52

vito commented Jan 8, 2021 •

edited

Loading

vito commented Jan 11, 2021

vito commented Jan 12, 2021

cpadgett-comcast commented Aug 4, 2021

support using local image tarballs with FROM #52

support using local image tarballs with FROM #52

Conversation

vito commented Jan 8, 2021 • edited Loading

vito commented Jan 11, 2021

vito commented Jan 12, 2021

cpadgett-comcast commented Aug 4, 2021

Example pipeline

build-docker-image.yaml

Dockerfile

pipeline.yaml

vito commented Jan 8, 2021 •

edited

Loading

`build-docker-image.yaml`

`Dockerfile`

`pipeline.yaml`